碎碎念

禮拜一上班就聽同事再說，filebeat stage環境有問題，
我還在禮拜一症候群阿~~
哎～SRE的宿命，先修吧

正文

一開始的問題很簡單，就output service設定錯誤，
無法連線。

改完後，換另一個錯誤。

Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open

google一下，就看到Elasticsearch 7.x node 開放 1000 個 shards 限制了。
上面連結是用curl 打網址直接改，下面是到dev Tools下指令，
所以有點不一樣。
到kibana上的dev tools加上這段，把1000的上限拉大。

PUT _cluster/settings
{
  "persistent": { 
    "cluster.max_shards_per_node": "3000" 
  }
}

ref. cluster-update-settings

Non-zero metrics in the last 30s

看filebeat裏面的訊息，一直出現這串。
關掉吧。

# If enabled, filebeat periodically logs its internal metrics that have changed
# in the last period. For each metric that changed, the delta from the value at
# the beginning of the period is logged. Also, the total values for
# all non-zero internal metrics are logged on shutdown. The default is true.
#logging.metrics.enabled: true

# The period after which to log the internal metrics. The default is 30s.
#logging.metrics.period: 30s

但在filebeat yaml中，是這樣的。

logging:
  metrics.enabled: false
  level: info
  to_files: true
  files:
    path: /var/log/filebeat
    name: filebeat
    keepfiles: 7
    permissions: 0644   

ref. Non-zero metrics in the last 30s: meaning